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The Academic Pathways Study (APS) of the NSE-funded Center for the Advancement of 
Engineering Education (CAEE) is a cross-university study that systematically examines how 
engineering students navigate their education, and how engineering skills and identity develop 
during their undergraduate careers. 

The APS has utilized a variety of methods including surveys, structured interviews, ethnographic 
methods, engineering design tasks, and academic transcripts to gain a broader and richer picture 
of students’ undergraduate engineering experiences. Of this portfolio of methods, a key 
component has been the survey, and through the collective work of the APS, two instruments 
have emerged - the Persistence In Engineering (PIE) survey and the Academic Pathways of 
People Learning Engineering Survey (APPLES). 


Building upon the PIE survey and the findings from the other APS methods, APPLES was 
designed to look at academic and professional persistence in two different cross-sectional 
populations of American undergraduate engineering students. The main objective of APPLES 
was to corroborate earlier findings from the APS on a larger scale and in particular, to explore 
the generalizability of findings from the PIE survey to engineering students at a greater number 
of American higher education institutions. 


Background of Survey Development 

The evolution of PIE to the APPLE surveys was guided 
by the need for comparability of findings while ensuring 
the integrity of each individual survey. The decisions 
about which PIE variables to include or exclude in the APPLES instruments were made 
according to methodological, institutional, and practical constraints. 


The main objective of APPLES 
was to corroborate earlier findings 
from the APS on a larger scale. 


The PIE survey was designed to identify and characterize the fundamental factors that influence 
students’ intentions to pursue an engineering degree over the course of their undergraduate 
career and upon graduation to practice engineering as a profession. Eirst administered in winter 
2003 of the students' freshman year, the PIE survey was deployed seven times to approximately 
160 students at four institutions with the final deployment in spring 2007 of their senior year 
(Longitudinal Cohort). 


The first administration of APPLES (APPLES 1) was conducted in April 2007 and was focused 
on the broader population of students at the same four core institutions (Broader Core Sample) 
who participated in the earlier PIE survey. These students had not previously taken the PIE 
survey and represented a second sample of students from these institutions, comparable to the 
Longitudinal Cohort. 

The second administration of APPLES (APPLES2) was conducted from January to March 2008 
with a carefully selected, stratified sample of students at 21 universities in the US (Broader 
National Sample). Sampling was done by institution using a stratified approach based on 
institutional characteristics. 

Constraints and Process of Survey Development 

In an ideal world, the team would simply have been able to administer the PIE survey instrument 
to both the Broader Core Sample and the Broader National Sample. However, cost was one 
constraint: the students in the Longitudinal Cohort were paid $175 annually to fill out the PIE 
surveys and participate in other data collection activities, while the APPLES respondents were 
compensated $4 to complete the survey. In addition, modification of the PIE survey's language 
as well as length was also necessary for a cross-sectional population. 

The obvious contrast in both time commitment and monetary incentive spurred the redesign and 
streamlining of the PIE instruments (which took students 20 to 40 minutes to complete) into a 
leaner survey that could be completed in approximately 10 minutes. The APPLES 1 instrument 
was created using the PIE survey as a foundation. In reducing the length of the survey, one major 
design consideration of APPLES was to still meet the research objective of testing the 
generalizability of preliminary results from the PIE survey. Other factors considered in the 
development of the APPLES instrument included scale reliability and validity, evidence of 
viability, and interest of the target audience in the findings. 

At their core, the PIE and APPLES instruments share a common set of variables representing the 
key concepts that researchers have suggested influence undergraduates’ persistence in the 
engineering major. (To view the table organizing the core variables according to the relevant 
APS research question category, see the full paper at the link below.) 

Prom PIE to APPLES 

Three factors guided the modifications and streamlining from the PIE survey to APPLES. The 
first involved checking the language of the questions, given that the targeted audience for the 
survey had now broadened to include freshmen, sophomores, juniors, seniors (traditional and 
those 5-1- years), as well as transfer and part-time students. The second set of changes focused on 
identifying which variables should be kept or eliminated. And, third, emerging findings form the 
other APS data collection methods were considered in order to identify additional factors about 
the undergraduate engineering experience that would influence persistence. 

Prom APPLES 1 to APPLES2 

One of the first steps in refining the APPLES 1 instrument for deployment to the national sample 
of 21 American institutions was to ensure that the demographic questions were appropriate and 
detailed enough to capture the diversity in institutions and student respondents. Changes made 
during the transition from APPLES 1 to APPLES 2 were minimal and were focused on bolstering 
internal reliability of several items. New items addressing psychological and behavioral 
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perspectives of intrinsic motivation were added after reviewing student responses to an open- 
ended question in APPLES 1 asking about their experiences in engineering education. 

One of the incentives for institutions to participate in the APPLES research study was the offer 
of an individualized campus report summarizing their students’ responses. As a result, the 
selection of questions for the APPLES instrument was also influenced by the interests of the 
anticipated audience for APPLES findings — deans, department chairs and faculty involved in 
ABET accreditation efforts, etc. — and by how these research findings might be interpreted and 
put into practice. 

Internal and external piloting of the survey questions was found to be essential both for checking 
the overall time to complete the survey online as well as the choice of items to include. In 
addition, features could be programmed into the survey tool to capture an estimated completion 
time for each potential survey item to guide decisions about items. 

Although results of these analyses are ongoing, the integration of findings from both quantitative 
and qualitative APS methods represents a valuable contribution and a useful model for future 
research. 
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